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SUMMARY 


The  papers  reviewed  here  deal  with  various  aspects  of  the  problem  of  system  identification 
when  all  the  observed  variables  are  contaminated  by  noise  errors.  Theoretical  issues  of 
iden^ktbiUty  and  practical  methods  for  developing  algorithms  to  estimate  parameters  of  a 
dynamic  system  are  considered.  The  total  least  squares  approach  and  the  Koopmans-Levin 
method  are  among  the  methods  discussed.  Both  these  methods  are  strongly  based  on  matrix 
singular  value  decomposition,  which  is  a  computationally  robust  numerical  technique.  A 
book,  which  is  a  compilation  qf  papers  originating  from  a  workshop  on  singular  value 
decomposition  and  signal  processing  is  included  in  the  references. 
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mtjmt^ucnoN 

of  tyotaa  parameters  using  noise-corrupted  data  is  of  fundamental  importance  in 
Iwcditem  of  iQretem  ideatificati<ni.  Many  identification  methods  are  based  on  the  assumption 
tiiMUt  ^Hiek  BO  input  noiw  in  the  system.  However,  this  is  not  true  in  most  practical  situations. 

A  Htting  tedoiique,  called  total  least  squares  (TLS),  can  be  used  for  parameter  estimation  when 
tiMre  k  soke  cm  all  the  data,  both  input  and  output.  TLS  has  a  long  history  in  the  statis¬ 
tical  Btecature  where  the  method  is  also  known  as  orthogonal  regression  or  errors-in-variables 
legraakoa.  la  recent  yours,  the  TLS  approach  has  been  applied  in  fields  outside  statistics. 

OdMT  meHiodB  are  also  available  for  system  identification  when  all  observed  variables  are  con- 
tamlBated  by  aoke  errors.  These  methods,  together  with  the  TLS  approach,  are  discussed  in 
the  reviews  in  Section  3.0. 

The  identification  problem  being  considered  is  that  of  estimating  the  parameters  in  a  transfer 
fianction  modd  of  a  dynamic  system.  This  problem,  given  in  its  errors-in-variables  form  is 

u(t)  =  tto(0  +  Au(t) 
y(t)  =  yo(0  +  Ay(t) 

Here  tio(t)  and  pio(t)  are  the  unmeasurable  noise-free  inputs  and  outputs,  u(t)  and  y(t)  the  noisy 
measured  inputs  and  outputs  while  Au(t)  and  Ap(t)  represent  all  stochastic  disturbances  to 
the  inputs  and  outputs,  respectively.  A{q~^)  and  i3(g“*)  are  polynomials  of  order  no  and  n6, 
respectively,  in  the  backward  shift  operator  and  have  the  following  form: 


A(q~^ )  =  1  +  oig"*  -i-  •  •  •  -I-  o„89“”“ 

The  backward  shift  operator  q~^  is  such  that 

g~‘v(t)  =  y(<-  1) 

2.0  INITIAL  REVIEW  OF  LITERATURE  SEARCHES 

Several  data  base  searches  were  carried  out  by  staff  of  the  ARL  library.  Preliminary  study  of 
these  literature  searches  identified  four  main  groups  of  papers.  These  are: 

1.  Peters  by  B  D  0  Anderson  and  his  co-authors.  Anderson  is  with  the  Department  of  Systems 
Ea^neoing,  Australian  National  University. 

2.  PiHMTS  by  S  Van  Huffd,  J  Vandewalle  and  others.  Many  of  these  authors  are  associated  with 
the  Kathoikke  Universitdt  Leuven. 

3.  Papers  and  a  bocdc  by  G  H  Golub  and  C  F  Van  Loan.  These  authors  are  from  the  Departments 
of  Computer  Scknoe  at  Stanford  and  Cornell  Universities  respectively. 

4.  A  pnpnr  by  K  V  Fbrnando  and  H  Nicholson.  These  authors  are  with  the  Department  of 
Control  Eni^Meriag,  University  of  Sheffield. 
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S.O  mSVlEW  OF  SELECTED  PAPERS 

Tl«  ptjpit*  ttlseted  £»  review  fall  into  two  broad  categories.  Papers  in  the  first  category  deal 
eAtli  tlw  pMblnn  of  estimating  the  parameters  in  a  transfer  function  model  of  a  dynamic  system. 
(Sift.  1,2,  3,  4,  10  and  11).  Papers  in  the  second  category  are  mainly  concerned  with  the 
watfcemitifil  and  computational  aspects  of  the  methods  used  for  system  identification.  (Refs. 
5, 0,7, 8  and  9). 

3.1  Review  of  papers  dealing  with  the  problem  of  estimating  the  parameters  in  a 
tnuwfto  fnaction  model  of  a  dynamic  system 

Refnrenoe  11  Soderstrom  T  (1081) 

The  purpose  oi  this  pi^er  is  to  investigate  when  and  how  a  dynamic  system  can  be  identified 
from  nedse-oorrupted  input-output  data.  After  brief  reviews  of  some  previous  publications,  the 
author  provides  a  formal  statement  of  the  identification  problem  under  discussion  and  a  review 
of  general  identifi^ility  properties. 

The  rest  of  the  paper  is  devoted  to  three  different  methods  for  identifying  systems  from  noisy 

data.  These  methods  are: 

1.  Spectral  analyns  (SA). 

2.  Corrdation  techniques  (CT). 

3.  Jdnt  output  approach  (JO). 

Details  these  methods  and  their  advantages  and  disadvantages  are  discussed  in  the  paper.  The 
CT  metiiod  can  be  interpreted  as  an  instrumental  variable  (IV)  method  using  delayed  inputs  as 
instruments. 

The  problem  being  considered  is: 


u{t)  =  w{t)  +  v{t) 
y{t)  =  z{i)  +  F{q~^)e{t) 

Here  to(t)  amd  x{t)  are  the  unmeasurable  noise-free  inputs  and  outputs,  while  u(f)  and  y{t)  are 
the  noisy  measured  inputs  auid  outputs. 

The  input  measurement  noise,  v((),  is  assumed  to  be  white  and  uncorrelated  with  the  output 
n<dse.  The  output  noise  is  assumed  to  describe  both  measurement  noise  and  process  noise  asting 
on  the  system.  It  is  assumed  that  the  output  noise  can  be  described  as  F{q~^  )e{t)  where  F{q~^) 
is  an  asymptotically  stable  filter  and  e(()  is  white  noise. 

In  smne  cases,  additional  assumptions  on  the  input  signal  are  used.  In  these  cases,  w{t)  is 

described  as 


w(<)  =  H(9->)/(0 

whsN  is  an  asymptoticadly  stable  filter  and  f{t)  is  white  noise,  uncorrdated  with  {e(*)} 

{*(•))•  The  vnriauice  matrices  of  e(f),  f{t)  and  v{t)  are  denoted  respectively  by  A*,  Af  and 
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lb  8hMtnte  tlie  properties  of  the  three  methods,  two  numerical  examples  were  considered. 
System  SI: 


G(q-^) 


l.Og  * 

1  -  0.8g-i 


1 


1  -  0.5g 


-1 


The  ncMse  variances  were  =  Ay  =  =  1.  The  undisturbed  signals  have  variances 

Ew{i)^  =  1.33  and  Ez(t)^  =  25.9.  Ten  different  realizations  of  500  data  pairs  were  generated. 

System  S2: 


G{q-^) 


l.Og-^  +  0.5g-^ 

1  -  l.5q-^  +  0.7g-2 


H{q-^) 


1 

1  -  0.9g-i 


The  nmse  variances  were  Aj  =  A^  =  A^  =  1.  The  undisturbed  signals  have  variances  Ew(t)^  = 
5.26  and  Ez{t)^  =  320.2.  Ten  different  realizations  of  200  data  pairs  were  generated. 

The  numerical  results  support  the  fdlowing  conclusions. 

The  modified  spectral  analysis  will  not  always  work,  since  estimated  spectral  densities  are  not 
necessarily  decomposable.  The  computational  requirements  are  far  higher  than  those  for  cor¬ 
relation  techniques  but  the  accuracy  (when  estimates  can  be  computed)  is  also  considerably 
better  than  CT.  The  main  merits  of  the  spectral  analysis  approach  is  to  conceptually  investigate 
the  identifiability  properties.  The  joint  output  approach  with  a  prediction  error  criteria  requires 
much  computer  time.  The  big  advantage  is  that  very  accurate  parameter  estimates  are  obtained. 


Roference  1  Fernando  K  V  and  Nicholson  H  (1985) 

The  objective  of  this  paper  is  to  develop  the  Koopmans- Levin  (KL)  method  of  identification 
of  linear  systems  using  singular  value  decomposition  (SVD).  As  Soderstrom  (Ref.  11)  has  not 
considered  the  KL  method,  the  results  here  complement  his  review  of  different  methods  of 
identification  in  the  presence  of  input  and  output  noise. 


In  this  paper,  three  algorithms  have  been  developed  for  identification  of  univariate  linear  discrete¬ 
time  systems.  The  technique  is  based  essentially  on  spectral  decomposition  of  a  covariance 
matrix  formed  using  input-output  data.  While  it  is  not  necessary  to  know  the  vauriances  of  the 
input  and  output  noise  explicitly,  it  is  assumed  that  the  ratio  of  these  variances  is  known. 

The  three  algorithms  are: 


1.  The  batch  or  direct  algorithm. 

2.  Sequential  updating  of  the  batch  algorithmic  solution.  This  algorithm  can  be  used  to  update 
the  batch  method  when  one  or  more  new  input-output  data  records  are  available. 
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3.  SwpMStial  estimation  using  an  information  matrix.  This  algorithm  is  based  on  an  information 
mtiislx  which  is  defined  as  the  inverse  of  the  covariance  matrix. 

'IHie  Mthcns  have  tested  the  batch  algorithm  extensively  on  the  two  numerical  examples  de* 
scifted  SdderstiSm  (Ref.  11).  They  have  also  tested  a  square-root  implementation  of  the 
iafomatioii  matrix  approach  using  the  same  examples. 

The  aathms  demonstrate  that  the  accuracy  of  the  estimates  using  the  KL  method  is  comparable 
with  that  of  the  joint  output  method,  and  superior  to  the  other  methods  described  by  Soderstrom 
(Sef.  11).  However,  numerical  requirements  of  the  KL  method  are  very  much  less  than  with 
the  joint  output  method. 

Referenee  10  Van  Huffel  S  and  Vandewalle  J  (1080) 

Ref.  10  considers  the  estimation  of  the  parameters  in  a  transfer  function  (TF)  model,  given  in 
its  emws-in-variables  form.  i.e. 


>1{9~M»o(<)  =  ^(9~*)«o(0  (la) 

u(t)  =  uo(0  +  A«(l)  (16) 

y(t)  =  yoit)  +  Ay(t)  (Ic) 

Here  ao(t)  and  )^(t)  are  the  unmeasurable  noise-free  inputs  and  outputs,  u(t)  and  y(t)  the  noisy 
measured  inputs  and  outputs  while  Au(t)  and  Ay{t)  represent  all  stochastic  disturbamces  to 
the  inputs  and  outputs,  respectively.  A{q~^)  and  B{q~^)  we  polynomials  of  order  no  and  n6, 
respectivdy,  in  the  backward  shift  operator  q~^  and  have  the  following  form; 

>l(q~l)  =  1 -1- aiq“l -f- •••  + o„o9“"“  (2a) 

5(rM  =  6w"‘+-+6„49-"*  (26) 

This  paper  evaluates  the  total  least  squares  (TLS)  approach  and  the  instrumental  variable  (IV) 
techniques  with  respect  to  parameter  estimation  in  the  TF  model  (1).  The  TLS  technique  is 
compared  with  several  IV  methods  for  estimating  the  parameters  in  the  TF  model  (1),  given 
only  the  normal  operating  input  and  output  of  the  system  over  a  limited  period  of  time.  It  is 
assumed  that  the  noise  sequences  added  to  the  noise-free  inputs  and  outputs  are  independent, 
discrete,  stationary  and  white  with  zero  mean  and  equed  variance. 

The  authors  conclude  that  TLS  is  especially  useful  in  the  estimation  of  the  parameters  of  a 
TF  model  for  short  sample  lengths,  when  the  outputs  and  possibly  the  inputs  are  disturbed  by 
white  nmse.  TLS  is  particularly  recommended  when  the  zeros  of  the  polynomial  A{z)  approach 
the  unit  circle  or  when  both  the  inputs  and  outputs  are  disturbed.  In  other  cases,  TLS  and  IV 
methods  give  comparable  results. 

The  authors  suggest  that  the  usefulness  of  the  TLS  method  may  be  questioned  for  large  sample 
lengths  when  only  output  noise  is  present.  The  computational  efficiency  is  then  an  importamt 
factor.  The  TLS  solution  involves  off-line  computations.  It  is  thus  computationally  not  so 
attractive  as  the  IV  method  which  exists  in  recursive  form,  allowing  good  on-line  estimation 
schones. 

The  TLS  method  is  simpler  to  use  since  the  IV  need  not  be  generated.  However,  the  TLS 
method  is  more  restrictive  since  the  IV  approach  allows  arbitrary  noise  models. 
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Sflft.  5-8  discuss  details  of  the  total  least  squares  problem. 

2  Anderson  B  D  O  (1985) 

S  Green  M  and  Anderson  B  D  O  (1986) 

Refurence  4  Anderson  B  D  O  and  Deistler  M  (1987) 

All  three  consider  errors-in-variables  identification  problems,  which  are  problems  where 

afi  observed  variables  are  contaminated  by  noise  errors.  Given  noisy  measurements  of  an  n- 
vector  process,  these  p^>er8  consider  such  questions  as:  “How  many  independent  (possibly 
hnear)  rdations  exist  among  the  non- noisy  components  of  the  process?”,  and  “What  is  the  set 
of  sudi  hnear  relations  for  which  the  data  are  compatible?”. 

In  the  main,  such  questions  have  been  posed  under  a  collection  of  standing  assumptions:  sta- 
tkmarity  of  the  underlying  processes,  linearity  of  the  underlying  relations,  and  availability  of 
covariance  data.  In  Ref.  2,  a  dynamic  problem  where  n  =  2  is  discussed.  In  Ref.  3,  dynamic 
problems,  where  the  measured  vector  of  dimension  2m  has  prescribed  partitioning  into  an  m- 
dimenskmal  input  vector  and  an  n-dimensional  output  vector,  are  discussed.  In  Ref.  4,  the 
authors  focus  on  the  case  n  =  3. 

In  Refii.  2  and  3,  the  authors  hypothesize  causality  of  a  transfer  function  (transfer  matrix  in 
Ref.  3)  appearing  in  a  dynamic  errors-in-variable  model  and  make  certain  other  reasonable 
assumptions.  Then  the  concern  of  these  two  papers  is  show  that  it  is  possible  to  parameterize 
the  class  of  transfer  functions  (transfer  matrices  in  Ref.  3)  consistent  with  the  available  data  in 
a  finite-dimensional  way. 

In  Ref.  2,  the  author  postulates  the  existence  of  three  real  random  sequences 
mutually  independent  and  stationary,  together  with  a  time-invariant  linear  system  defined  by 
a  real  bounded  linear  causal  convolution  operator  >  0}  mapping  (zK)  into  a  sequence 

according  to 

k 

-oo 

The  processes  i^ib}  available  for  measurement,  but  rather  one  can  measure  for 

k  €  (-00,00) 


Xk  =  xl  +  u*  (3.2a) 

y*  =  yfc  +  (3-26) 

Attention  is  confined  to  Gaussian  processes.  The  concern  is  not  to  identify  a  unique  {wj^},  but 
to  characterize  the  class  of  which  fit  the  data. 

The  power  spectrum  matrix  of  [z  y\ 

a'yt(u;) 

is  termed  the  standard  data. 

The  standing  assumptions  are  set  out  below. 

The  random  sequences  {x^},  and  {vj^}  are  independent,  zero  mean  processes  of  which 

measurements  are  available  according  to  (3.2).  Further,  they  are  regular  processes,  so  that  each 

possesses  a  bounded  spectral  density. 
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b  aditttioii,  it  is  assumed  that  the  process  is  generated  by 

k 

— oo 

udiere  is  a  sero  mean,  stationary,  white  noise  sequence  and  {wj^,  i;  >  0}  is  a  causal  impulse 

teapooae  satisfying 

00 

<  00  for  some  p>  1. 

k=0 

It  is  also  assumed  that  the  impulse  response  ^  0}  satisfies  the  strengthened  stability 

requiranent 

oo 

p*|tufc|  <  00  for  some  p>  1. 

k=0 


It  is  supposed  that  W(z)  is  the  transfer  function  associated  with  the  sequence  defined  by 


OO 

0 

Here,  x  is  the  backward  shift  operator,  and  the  standing  assumptions  ensure  that  W{z)  is 
analytic  inside  |s|  <  p. 

The  problem  is  to  define,  to  the  fullest  extent  possible,  what  W{x)  is,  under  the  standing 
assumptions  and  given  the  standard  data. 


Under  the  standing  assumptions  together  with  the  standard  data,  the  following  relationships 
are  derived. 


and 


arg 


(Tn(w). 


=  orp[iy(e^‘^)]  =  arg 


^yy{^) 


(3.3a) 


Then  the  remainder  of  the  paper  is  devoted  to  constructing  W(e^'^)  satisfying  (3.3)  and  param¬ 
eterizing  the  solution  set  of  (3.3).  The  key  idea  is  to  construct  the  magnitude  of  W{e^'^)  from 
the  phase,  which  is  known  from  (3.3a),  by  using  a  formula  from  analytic  function  theory  relating 
the  real  and  imaginary  parts  of  analytic  functions. 

The  information  in  (3.3)  coupled  with  an  assumption  that  1^(6^*^)  is  causal  determines  the 
number  of  zeros  of  W(e^'^)  inside  the  unit  circle.  If  there  are  N  such  zeros,  the  solution  set  is 
shown  to  be  an  fir  -1- 1  parameter  family.  The  simplest  case  is  that  of  minimum  phase  W{z)  — 
those  where  there  are  no  zeros  inside  the  unit  circle.  In  the  case  of  minimum  phase  transfer 
functions,  the  parameter  is  a  scalar. 

The  work  of  Anderson  discussed  above  has  been  extended,  by  Green  and  Anderson,  to  mul¬ 
tivariable  nrors-in-variables  models  in  Ref.  3.  In  the  multivariable  case,  it  is  still  possible  to 
obtain  formulas  analagous  to  (3.3).  However,  there  are  difficulties  involved  in  applying  the  scalar 


^yyW) 

Oyxiy) 


(3.36) 
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soltttioii  technique  outlined  above.  The  scalar  technique  proceeds  from  phase  information,  but 
what  is  the  phase  of  a  matrix? 

The  solution  technique  presented  in  Ref.  3  is  based  on  the  factorization  of  matrix  valued 
functions,  a  special  case  of  which  is  the  better  known  spectral  factorization.  The  factorization 
theory  is  wdl  developed  and  is  used  extensively  in  the  theory  of  integral  equations.  A  summary 
of  matrix  factorization  theory  is  presented  in  Section  III  of  the  paper.  Then  in  Section  IV,  the 
authors  completely  characterize  the  solution  set  of  the  multivariable  errors-in- variable  problem 
in  terms  of  the  factors  of  the  cross-spectrum  matrix 

In  Ref.  4  the  authors  deal  with  two  different,  though  related,  problems.  The  first  is  a  complex 
veruon  of  a  static  errors-in- variables  problem.  This  problem  has  been  analysed  in  detail  and  a 
solution  has  been  ^ven  for  the  case  of  three  scalar  variables.  In  the  second  problem,  the  authors 
have  searched  for  causal  solutions  of  a  dynamic  three  variable  problem.  A  checkable  necessary 
condition  for  a  solution  to  exist  has  been  exhibited,  and  an  indication  given  of  how  a  solution 
might  (numerically)  be  obtained. 

3.2  Review  of  papers  concerned  with  the  mathematical  and  computational  aspects 
of  the  methods  used  for  system  identification 

Reference  8  Golub  G  H  and  Van  Loan  C  F  (1980) 

In  this  paper  a  singular  value  decomposition  analysis  of  the  total  least  squares  (TLS)  problem 
is  presented  and  an  algorithm  for  solving  the  TLS  problem  is  proposed. 

Reference  5  Van  HufFel  S  and  Vandewalle  J  (1988) 

The  authors  indicate  that  in  some  TLS  problems,  called  nongeneric,  the  algorithm  proposed  by 
Golub  and  Van  Loan  (Ref.  8)  fails  to  compute  a  finite  TLS  solution.  This  paper  generalizes  their 
TLS  computations  in  order  to  solve  these  nongeneric  TLS  problems.  An  algorithm  is  presented 
which  includes  the  proposed  generalizations. 

Reference  6  Van  Huffel  S  and  Vandewalle  J  (1989) 

Total  least  squares  (TLS)  is  one  method  of  solving  overdetermined  sets  of  linear  equations 
AX  «  B  that  is  appropriate  when  there  are  errors  in  both  the  observation  matrix  B  and  the 
data  matrix  A.  In  some  linear  parameter  estimation  problems,  some  of  the  variables  may  be 
observed  without  error.  This  implies  that  some  of  the  columns  of  A  are  assumed  to  be  known 
exactly.  To  deal  with  this  case,  a  computationally  efficient  and  numerically  reliable  Generalized 
TLS  algorithm  GTLS,  based  on  the  Generalized  SVD  (GSVD),  is  developed. 

Reference  7  Van  Huffel  S  and  Vandewalle  J  (1989) 

This  paper  compares  least  squares  ( LS)  and  total  least  squares  (TLS)  methods  for  solving  overde¬ 
termined  sets  of  linear  equations,  AX  «  B,  arising  in  linear  parameter  estimation  problems. 
The  authors  examine,  via  simulations,  how  perturbations  on  both  the  data  matrix  A  and  the 
observation  matrix  B  affect  the  accuracy  of  the  TLS  and  LS  problem. 

Reference  9  Deprettere  F,  editor  (1988) 

This  book  owes  its  origins  to  the  Workshop  on  SVD  AND  SIGNAL  PROCESSING,  which  was 
held  at  the  Les  Houches  summer  school  for  Physics,  September  21-23,  1987. 

A  selection  of  papers  is  presented,  dealing  with  matrix  singular  value  decomposition,  applications 
of  SVD  in  signal  processing  and  system  identification  and  related  algorithms  and  implementation 
architectures. 


7 


4.0  CONCLUDING  REMARKS 


The  p&pers  reviewed  in  Section  3.0  discuss  both  theoretical  issues  of  identihability  and  prac¬ 
tical  methods  for  system  identification  when  all  observed  variables  are  contaminated  by  noise 
errors.  Details  are  provided  for  a  number  of  different  methods  of  estimating  the  parameters  in 
a  transfer  function  model  of  a  dynamic  system.  Advantages  and  disadvantages  of  each  method 
are  discussed,  and  in  some  cases  comparisons  between  methods  have  been  carried  out.  Lists  of 
references  in  the  papers  reviewed  provide  further  avenues  for  information  as  required. 
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